SICS at iCLEF 2008: User Confidence and Satisfaction Tentatively Inferred from iCLEF Logs

This paper gives a brief description of some initial experiments performed at SICS using the interactive image search query logs provided for participants in the interactive track of CLEF. The SICS experiments attempt to establish whether user confidence and trust in results can be related to logged behaviour. SICS has participated in this year’s iCLEF cycle mainly with an eye on future experimental settings to work on measurement of searcher trust and confidence in the search process and its results, in keeping with previous experimental studies performed at SICS [2, 1, e.g.]. SICS has used the Flickling interface [3] and the logs delivered by it to study how searcher actions can be interpreted as exponents of user confidence. Variables under consideration for this purpose can be indirect, such as length of interaction, time spent on query formulation, and other measures which require non-trivial interpretation during the analysis phase. Other variables can be more direct, in that they more clearly indicate competence or confidence on the part of the searcher, such as observed edits and additions made to the user dictionaries by the searcher or the number of times a query is reformulated. A confident searcher can be assumed to be more likely to enter edits into the user dictionary and not to reformulate queries to the same extent. In Table 1 such measures are tabulated, with a distinction between sequences of actions that ultimately provide a successfully identified target image and sequences which terminate by the searcher giving up requesting another target. We find that while the differences given are statistically significant by Mann Whitney’s U , they are less sizeable than might have been expected, and in some cases do not conform to expectation as to their direction: one would expect unsuccessful sequences use more hints than successful sequences, e.g. Better measures of user confidence in their actions should be measurable somehow, but these logs give little purchase for this type of analysis. For next year’s cycle we plan to investigate whether user actions can be logged to include their sense of satisfying a search — by indicating that a found image might be good enough even if not identical to the target image. Number of sessions Ratio of reformulated Number of with adds to dictionary queries hints taken Successful search sequences 0.087 0.77 2.3 Unsuccessful search sequences 0.090 0.74 1.7 Table 1. Indications of user confidence.